Activity situation analysis apparatus, activity situation analysis system, and activity situation analysis method

ABSTRACT

Position information on every moving object is acquired from a moving image of a monitoring area, and activity information during every unit time is acquired from the position information on every moving object. Conditions of an observation period of time are set according to a user input operation, the observation period of time is controlled in accordance with the conditions of the observation period of time, and the activity information during every unit time is aggregated during the observation period of time to acquire the activity information during the observation period of time. An activity map image is generated from the activity information during the observation period of time, and the activity map image and the moving image of the monitoring area are generated and output at every predetermined point in time.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an activity situation analysisapparatus, an activity situation analysis system, and an activitysituation analysis method, in each of which an activity situation of amoving object in a monitoring area is observed, an activity map imagethat results from visualizing the activity situation of the moving bodyis generated based on a result of the observation, and the generatedactivity map image is displayed on a display device.

2. Description of the Related Art

In a store such as a convenience store, a plan for an improvement instore management, specifically, an improvement plan for a review oftypes of commodities at every salesroom, a method of displaying thecommodities at every salesroom, and the like, are considered based onanalysis relating to a behavior of a customer within the store, but theanalysis is useful for planning to achieve an improvement in customersatisfaction or efficient store management and for improving a profit ofand a sales volume of the store.

On the other hand, in the store such as the convenience store, amonitoring system, in which a situation within the store is monitoredusing a moving image that is captured by a camera which is installed tocapture a moving image of the inside of the store, has come into wideuse, and when an information processing apparatus is caused to performthe analysis relating to the behavior of the customer within the storeusing the moving image that is captured by the camera, a task ofstudying the improvement plan for store management can be efficientlyperformed.

As a technology that performs the analysis relating to the behavior ofthe person using the moving image of the camera, in the related art, atechnology is known that acquires the information relating to theactivity situation of the person in a monitoring area from plural movingimages of the camera and generates an activity map image that resultsfrom visualizing the activity situation (refer to Japanese PatentUnexamined Publication No. 2009-134688).

With this technology, the activity map image is set to be color-codedinto the shape of a contour according to the extent of person activity,and is set to be displayed in a state of being superimposed onto alayout drawing of the monitoring area. Furthermore, a technology isknown in which the monitoring area is divided into multiple blocks andthat detects the extent to which the person stays in every block (referto Japanese Patent Unexamined Publication No. 2011-248836). With thistechnology, a value (score) indicating the extent to which the personstays is set to be output in every block.

The moving image of the monitoring area is displayed and thus the usercan know an actual situation within the monitoring area. However, whenscreen display through which how the activity situation of the personchanges can be known is performed while the moving image of themonitoring area is in the middle of being displayed in this manner, theactivity situation of the person in the monitoring area can be suitablyknown.

Particularly, a configuration is desirable in which an activitysituation within the monitoring area at an attention-receiving point intime (a current point in view in real time display) and an activitysituation of the person during an attention-receiving period of time canbe known at the same time.

However, in Japanese Patent Unexamined Publication No. 2009-134688 andJapanese Patent Unexamined Publication No. 2011-248836, the user canknow a distribution situation of the activity extent of the personwithin the monitoring area, that is, at which area the person is activeto which extent, but there is a problem that how the activity situationof the person changes cannot be completely known without taking intoconsideration the desire described above, while the moving image of themonitoring area is in the middle of being displayed.

Furthermore, the activity situation of the person can be known with theactivity map image, but when an analysis form of the activity situationthat is to be applied when the activity map image is generated,particularly, conditions relating to the observation period of time thatis to be applied when the activity information that is a source of theactivity map image is generated can be set to be variously changed, theactivity situation of the person in the monitoring area can be knownfrom various points of view, thereby improving user's convenience.Because of this, a configuration is desirable in which customizationthat variously changes the analysis form of the activity situationaccording to a user need can be easily performed.

SUMMARY OF THE INVENTION

An activity situation analysis apparatus according to an aspect of theinvention observes an activity situation of a moving object in amonitoring area, generates an activity map image that results fromvisualizing the activity situation of the moving object based on aresult of the observation, and displays the generated activity map imageon a display device. The activity situation analysis apparatus includes:a position information acquirer that detects the moving object from amoving image of the monitoring area and acquires position information onevery moving object; an activity information acquirer that acquiresactivity information based on the position information on every movingobject; an observation condition setter that sets conditions relating toan observation period of time for observing activity situation of themoving object according to a user input operation; anobservation-period-of-time controller that controls the observationperiod of time in accordance with the conditions relating to theobservation period of time that is set by the observation conditionsetter; an activity information aggregator that aggregates the activityinformation based on the observation period of time that is controlledby the observation-period-of-time controller, and acquires the activityinformation during the observation period of time; an activity map imagegenerator that generates the activity map image based on the activityinformation during the observation period of time; and an outputcontroller that generates a monitoring moving image including theactivity map image and the moving image of the monitoring area at everypredetermined point in view, and outputs the generated monitoring movingimage to the display device.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a configuration diagram of an entire activity situationanalysis system according to a first embodiment;

FIG. 2 is a top-view diagram for describing a store layout and aninstallation situation of a camera;

FIG. 3A is an explanatory diagram for describing an outline ofprocessing that is performed on a PC;

FIG. 3B is an explanatory diagram for describing an outline of theprocessing that is performed on the PC;

FIG. 4 is an explanatory diagram for describing processing thatgenerates a monitoring moving image;

FIG. 5A is an explanatory diagram for describing an observation periodof time that is to be applied when activity information that is a sourceof an activity map image is generated;

FIG. 5B is an explanatory diagram for describing the observation periodof time that is to be applied when the activity information that is thesource of the activity map image is generated;

FIG. 6A is an explanatory diagram for describing the observation periodof time that is to be applied when the activity information that is thesource of the activity map image is generated;

FIG. 6B is an explanatory diagram for describing the observation periodof time that is to be applied when the activity information that is thesource of the activity map image is generated;

FIG. 7 is an explanatory diagram for describing processing thatgenerates the activity information that is the source of the activitymap image;

FIG. 8 is a functional block diagram illustrating a schematicconfiguration of the PC;

FIG. 9 is an explanatory diagram illustrating a monitoring screen thatis displayed on a monitor;

FIG. 10A is an explanatory diagram illustrating a display point-in-timeand observation period-of-time designation section that is illustratedin FIG. 9;

FIG. 10B are explanatory diagrams illustrating the display point-in-timeand observation period-of-time designation section that is illustratedin FIG. 9;

FIG. 11 is an explanatory diagram for describing a different example ofthe display point-in-time and observation period-of-time designationsection;

FIG. 12A is an explanatory diagram illustrating a transition situationof the display point-in-time and observation period-of-time designationsection;

FIG. 12B is an explanatory diagram illustrating the transition situationof the display point-in-time and observation period-of-time designationsection;

FIG. 13A is an explanatory diagram illustrating the transition situationof the display point-in-time and observation period-of-time designationsection in different examples of a display point in time and theobservation period of time;

FIG. 13B is an explanatory diagram illustrating the transition situationof the display point-in-time and observation period-of-time designationsection in the different examples of the display point in time and theobservation period of time;

FIG. 14 is an explanatory diagram illustrating a processing conditionsetting screen that is displayed on the monitor;

FIG. 15 is an explanatory diagram illustrating a different example ofthe processing condition setting screen that is illustrated in FIG. 14;

FIG. 16A is an explanatory diagram illustrating the activity map imagethat is an area division type according to a second embodiment;

FIG. 16B is an explanatory diagram illustrating the activity map imagethat is the area division type according to the second embodiment; and

FIG. 17 is a functional block diagram illustrating schematicconfigurations of the camera and the PC in an activity situationanalysis system according to a third embodiment.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

A main object of the present invention, which is made to resolve theproblems with the technologies in the related art, is to provide anactivity situation analysis apparatus, an activity situation analysissystem and an activity situation analysis method, each of which isconfigured in such a manner that a user can easily know how an activitysituation of a moving object changes while a moving image of amonitoring area is being displayed and customization that variouslychanges an analysis form of the activity situation according to a userneed can be easily performed.

According to a first invention, there is provided an activity situationanalysis apparatus that observes an activity situation of a movingobject in a monitoring area, generates an activity map image thatresults from visualizing the activity situation of the moving objectbased on a result of the observation, and displays the activity mapimage on a display device, the activity situation analysis apparatusbeing configured to include; a position information acquirer thatdetects the moving object from a moving image of the monitoring area andacquires position information on every moving object; an activityinformation acquirer that acquires activity information based on theposition information on every moving object; an observation conditionsetter that sets a condition relating to an observation period of timefor observing the activity situation of the moving object according to auser input operation; an observation-period-of-time controller thatcontrols the observation period of time in accordance with theconditions relating to the observation period of time that is set by theobservation condition setter; an activity information aggregator thataggregates the activity information based on the observation period oftime that is controlled by the observation-period-of-time controller andacquires activity information during the observation period of time; anactivity map image generator that generates the activity map image basedon the activity information during the observation period of time; andan output controller that generates a monitoring moving image includingthe activity map image and the moving image of the monitoring area atevery predetermined point in time and outputs the generated monitoringmoving image to the display device.

Accordingly, because the monitoring moving image including the activitymap image and the moving image of the monitoring area is generated at apredetermined point in time and the generated monitoring moving image isoutput, while the moving image of the monitoring area is in the middleof being displayed, the user can easily know how the activationsituation of the moving object changes. Furthermore, because theobservation period of time is controlled in accordance with theconditions that are set according to the user input operation, thecustomization that variously changes the analysis form of the activitysituation according to the user's need can be easily performed.

Furthermore, according to a second invention, the output controller maybe configured to generate the monitoring moving image that results fromsuperimposing the activity map image on the moving image of themonitoring area and to output the generated monitoring moving image.

Accordingly, the user can immediately know an actual situation of andthe activity situation of the moving object in an attention-receivingarea within the monitoring area.

Furthermore, according to a third invention, theobservation-period-of-time controller may be configured to control anobservation period of time in such a manner that, according to theprogress of a display point in time of the moving image of themonitoring area, the observation period of time is moved following thedisplay point in time.

Accordingly, with the activity map image, the user can know the activitysituation of the moving object during a period of time that is based onthe display point in time of the moving image of the monitoring area.When a display point in time progresses, the observation period of timeis moved, and the activity information during the observation period oftime is updated, but at this time, processing is only performed thatexcludes and integrates activity information that is equivalent to adifference between a new observation period of time and the previousobservation period of time from and into the activity information duringthe previous observation period of time, and thus activity informationduring the new observation period of time can be acquired. Because ofthis, an arithmetic operation load can be reduced.

Furthermore, according to a fourth embodiment, theobservation-period-of-time controller may be configured to control theobservation period of time in such a manner that, based on a settinglength of the observation period of time that is set by the observationcondition setter, a period of time that is by the setting length of theobservation period of time earlier than the display point in time of themoving image of the monitoring area is the observation period of time.

Accordingly, with the activity map image, the user can know the activitysituation of the moving object during the period of time immediatelybefore the moving image of the monitoring area.

Furthermore, according to a fifth invention, theobservation-period-of-time controller may be configured to control theobservation period of time in such a manner that a period of time from adisplay starting point in time to the display point in time is theobservation period of time until an elapsed period of time from thedisplay starting point in time of the moving image of the monitoringarea to the display point in time reaches the setting length of theobservation period of time.

Accordingly, because past information at an earlier time than thedisplay starting point in time of the moving image of the monitoringarea is not included in the activity map image, the user can suitablyknow the activity situation of the moving object for a period of time atan attention-receiving point in time or later.

Furthermore, according to a sixth invention, theobservation-period-of-time controller may be configured to control theobservation period of time in such a manner that, based on a settingrange of the observation period of time, which is set by the observationcondition setter, within the setting range of the observation period oftime, a period of time from the display starting point in time of themoving image of the monitoring area, which is a starting point of theobservation period of time, to a display point in time is theobservation period of time.

Accordingly, with the activity map image, the user can know the activitysituation of the moving object that occurs after displaying of themoving image of the monitoring area is started. When the display pointin time progresses, the observation period of time is moved, and theactivity information during the observation period of time is updated,but at this time, activity information that is equivalent to adifference between the previous observation period of time and a newobservation period of time is only integrated into the activityinformation during the previous observation period of time, and thusactivity information during the new observation period of time can beacquired. Because of this, an arithmetic operation load can be reduced.

Furthermore, according to a seventh invention, the observation conditionsetter may be configured to set the length of or the range of theobservation period of time according to the user input operation thatarbitrarily designates the length of or the range of the observationperiod of time as conditions of the observation period of time.

Accordingly, because the user can arbitrarily designate the length of orthe range of the observation period of time, a customization attributethat changes the analysis form of the activity situation can beimproved.

Furthermore, according to an eighth invention, the output controller maybe configured to generate and output in real time the monitoring movingimage including the activity map image and a current moving image of themonitoring area.

Accordingly, the user can browse through the activity map image and thecurrent moving image of the monitoring area in real time.

Furthermore, according to a ninth invention, the activity situationanalysis apparatus may be configured to further include a unit timesetter that sets a unit time relating to the acquisition of the activityinformation according to the user input operation, and in the activitysituation analysis apparatus, the activity information acquirer may beconfigured to acquire the activity information during every unit timebased on the unit time that is set by the unit time setter.

Accordingly, because the user can designate the unit time, thecustomization attribute that changes the analysis form of the activitysituation can be improved.

Furthermore, according to a tenth invention, the activity informationmay be configured to include information relating to at least either ofthe number of stays that is the number of moving objects that stay in anobservation area that is set to be within the monitoring area and astaying time that is a time for which the moving object stays within theobservation area.

Accordingly, because with the activity, the user can know the number ofstays or the staying time in the observation area using the activity mapimage, the user can suitably know the activity situation (a stayingsituation) of the moving object.

Furthermore, according to an eleventh invention, there is provided anactivity situation analysis system that observes an activity situationof a moving object in a monitoring area, generates an activity map imagethat results from visualizing the activity situation of the movingobject based on a result of the observation, and displays the activitymap image on a display device, the activity situation analysis systembeing configured to include; a camera that captures a moving image ofthe monitoring area; and multiple information processing apparatuses, inwhich any one of the multiple information processing apparatusesincludes a position information acquirer that detects the moving objectfrom the moving image of the monitoring area and acquires positioninformation on every moving object, an activity information acquirerthat acquires activity information based on the position information onevery moving object, an observation condition setter that sets acondition relating to an observation period of time for observing theactivity situation of the moving object according to a user inputoperation, an observation-period-of-time controller that controls theobservation period of time in accordance with the conditions of theobservation period of time that is set by the observation conditionsetter, an activity information aggregator that aggregates the activityinformation based on the observation period of time that is controlledby the observation-period-of-time controller and acquires activityinformation during the observation period of time, an activity map imagegenerator that generates the activity map image based on the activityinformation during the observation period of time, and an outputcontroller that generates a monitoring moving image including theactivity map image and the moving image of the monitoring area at everypredetermined point in time and outputs the generated monitoring movingimage to the display device.

Accordingly, as is the case with the first invention, while the movingimage of the monitoring area is being displayed, the user can easilyknow how the activity situation of the moving object changes, and caneasily perform the customization that variously changes the analysisform of the activity situation according to the user's need.

Furthermore, according to a twelfth invention, there is provided anactivity situation analysis system that observes an activity situationof a moving object in a monitoring area, generates an activity map imagethat results from visualizing the activity situation of the movingobject based on a result of the observation, and displays the activitymap image on a display device, the activity situation analysis systembeing configured to include; a camera that captures a moving image ofthe monitoring area; and an information processing apparatus, in whichany one of the camera and the information processing apparatus includesa position information acquirer that detects the moving object from themoving image of the monitoring area and acquires position information onevery moving object, an activity information acquirer that acquiresactivity information based on the position information on every movingobject, an observation condition setter that sets a condition relatingto an observation period of time for observing the activity situation ofthe moving object according to a user input operation, anobservation-period-of-time controller that controls the observationperiod of time in accordance with the conditions of the observationperiod of time that is set by the observation condition setter, anactivity information aggregator that aggregates the activity informationbased on the observation period of time that is controlled by theobservation-period-of-time controller and acquires activity informationduring the observation period of time, an activity map image generatorthat generates the activity map image based on the activity informationduring the observation period of time, and an output controller thatgenerates a monitoring moving image including the activity map image andthe moving image of the monitoring area at every predetermined point intime and outputs the generated monitoring moving image to the displaydevice.

Accordingly, as is the case with the first invention, while the movingimage of the monitoring area is being displayed, the user can easilyknow how the activity situation of the moving object changes, and caneasily perform the customization that variously changes the analysisform of the activity situation according to the user's need.

Furthermore, according to a thirteenth invention, there is provided anactivity situation analysis method of causing an information processingapparatus to perform processing that observes an activity situation of amoving object in a monitoring area, generates an activity map image thatresults from visualizing the activity situation of the moving objectbased on a result of the observation, and displays the activity mapimage on a display device, the activity situation analysis method beingconfigured to include: detecting the moving object from a moving imageof the monitoring area and acquiring position information on everymoving object; acquiring activity information based on the positioninformation on every moving object; setting a condition relating to anobservation period of time for observing the activity situation of themoving object according to a user input operation; controlling theobservation period of time in accordance with the conditions relating tothe observation period of time that is set in the setting of thecondition relating to the observation period of time; aggregating theactivity information based on the observation period of time that iscontrolled in the controlling of the observation period of time andacquiring activity information during the observation period of time;generating the activity map image based on the activity informationduring the observation period of time; and generating a monitoringmoving image including the activity map image and the moving image ofthe monitoring area at every predetermined point in time and outputtingthe generated monitoring moving image to the display device.

Accordingly, as is the case with the first invention, while the movingimage of the monitoring area is being displayed, the user can easilyknow how the activity situation of the moving object changes, and caneasily perform the customization that variously changes the analysisform of the activity situation according to the user's need.

Embodiments of the present invention will be described below referringto the drawings.

First Embodiment

FIG. 1 is a configuration diagram of an entire activity situationanalysis system according to a first embodiment. The activity situationanalysis system is established for a retail chain store such as aconvenience store, and includes camera 1, recorder (a moving imagestorage) 2, and PC (an activity situation analysis apparatus) 3.

Camera 1 is installed at a suitable place within a store (a facility). Amoving image of the inside of the store is captured by camera 1, and themoving image obtained by this is stored in recorder 2.

Input device 6 on which a user such as a store manager performs variousinput operations, such as a mouse, and monitor (a display device) 7 onwhich a monitoring screen is displayed are connected to PC 3. With themonitoring screen that is displayed on monitor 7, the user can browsethrough moving images of the inside of the store, which are captured bycamera 1, in real time and can browse through past moving images of theinside of the store, which were recorded in recorder 2.

Furthermore, camera 1, recorder 2, and PC 3 are installed in each of themultiple stores, and PC 11 is installed in a head office that generallymanages the multiple stores. On PC 11, the moving images of the insideof the store that are captured with camera 1 can be browsed through inreal time, the past moving images of the inside of the store that wererecorded in recorder 2 can be browsed through, and thus, a situation ofthe inside of the store can be checked in the head office.

According to the present embodiment, PC 3 installed in the store isconfigured as an activity situation analysis apparatus that performsanalysis relating to an activity situation of a customer within thestore. Analysis result information generated on PC 3 can be browsedthrough on PC 3 by a user on the store side, for example, a storemanager, is transmitted to PC 11 installed in a head office and thus canalso be browsed through on PC 11 by a user on the head office side, forexample, a supervisor who provides guidance or a proposal to each of thestores within a district of which the supervisor is in charge.Furthermore, PCs 3 and 11 are configured as browsing apparatuses inwhich the analysis result information is browsed through.

Next, a store layout and an installation state of camera 1 aredescribed. FIG. 2 is a top-view diagram for describing the store layoutand the installation situation of camera 1.

In the store, an entrance door, display shelves, a check-out counter,and the like are installed. The display shelves are installed in a stateof being categorized according to types of commodities such as aJapanese box lunch, a PET bottle, and a rice ball. A customer enters thestore through the entrance door, and moves along a passage between thedisplay shelves within the store. When the customer finds a commoditythat he/she wants to purchase, he/she picks it up and moves to aregister counter. After making payment on account (paying for thecommodity) at the register counter, the customer leaves the storethrough the entrance door.

Furthermore, in the store, multiple cameras 1 that capture moving imagesof the inside (the monitoring area) of the store are installed. Thecamera is installed in a suitable position on the ceiling of the store.Particularly, in an example that is illustrated in FIG. 2, as camera 1,an omnidirectional camera with a fish-eye lens that has a photographingrange of 360 degrees is adopted, and with this camera 1, images ofpersons that enter and leave the store through the entrance door, imagesof persons that are present within the store, and moving images ofcommodities and the like on the display shelves can be captured.

Next, an outline of processing that is performed on PC 3 that isillustrated in FIG. 1 is described. FIGS. 3A and 3B are explanatorydiagrams for describing the outline of the processing that is performedon PC 3.

As illustrated in FIG. 3A, persons who stop in front of the displayshelves and determine the merits of each commodity, persons who shoparound to find the commodities that they want to buy, or the like areshown in a moving image that is captured by camera 1. Here, the personswho stop in front of the display shelves to determine the merits of thecommodities do not move in front of the display shelves for apredetermined period of time or longer, and thus are in a state wherethey stay in front of the display shelves. The activity situation(staying situation) of the person in front of the display shelfindicates the extent to which the customer is interested in thecommodity on the display shelf, and is useful for considering animprovement plan for store management, that is, an improvement plan fora review of types of commodities on every sales room, a method ofdisplaying the commodities, and the like.

Here, according to the present embodiment, in order to know the extentto which the customer is interested in a salesroom that the user such asthe store manager pays attention to, specifically in the display shelffor commodities that the user pays attention to, an observation area isset to be in an area where a passage and the like in front of thedisplay shelf for commodities that the user pays attention to is shown,and the activity information relating to an activity situation of aperson within the observation area, specifically, the number of stays(the number of staying persons), that is, the number of persons stayingwithin the observation area, and staying time, that is, time for whichthe person within the observation area stays are acquired. The activitymap image that results from visualizing the activity information isgenerated, and the monitoring moving image that results fromsuperimposing the activity map image onto the moving image of themonitoring area, as illustrated in FIG. 3B, is generated and isdisplayed. With the activity map image, the extent to which the customeris interested in the salesroom corresponding to the observation area(the display shelf for commodities) can be known.

In an example that is illustrated in FIG. 3A, 3 observation areas A1 toA3 are set to be in a moving image, and as illustrated in FIG. 3B,activity map images H1 to H3 are displayed on positions corresponding toobservation areas A1 to A3.

In the activity map image, a numerical value of activity information isexpressed by changing the display form (characteristics of the movingimage), specifically, by at least one among the display elements thatare the size and the color (a hue, intensity, or the like).Particularly, in an example that is illustrated in FIG. 3B, the activitymap images H1 to H3 take the shape of an ellipse. The staying time isexpressed with the intensity of a filling color of the ellipse, and thenumber of stays (the number of staying persons) is expressed with sizesof the activity map images H1 to H3. The longer the staying time, themore intense the colors of the activity map images H1 to H3 is. Thegreater the number of stays, the greater the sizes of the activity mapimages H1 to H3.

According to the present embodiment, because the activity map image isset to be transmissive, a state where the moving image of the monitoringarea is transmissive and seen is attained.

Next, processing that generates the monitoring moving image that isillustrated in FIG. 3B is described. FIG. 4 is an explanatory diagramfor describing the processing that generates the monitoring movingimage.

According to the present embodiment, the position information on everyperson at each point in time is acquired from a moving image (a frame)that is captured at each point in time, temporal statistical processingis performed on the position information on every person at each pointin time within a predetermined observation period of time (for example,30 minutes, one hour, or the like), and thus the pieces of activityinformation (the staying time and the number of stays) relating to theactivity situation of the person are acquired. The activity map imagethat results from visualizing the activity information is generated, andthe monitoring moving image that results from superimposing the activitymap image onto the moving image of the monitoring area at a displaypoint in time is generated.

In the monitoring moving image, according to the progress of the displaypoint in time, the moving image of the monitoring area changes frommoment to moment. On the other hand, there is a case where theobservation period of time changes according to the progress of thedisplay point in time and a case where the observation period of timedoes not change according to the progress of the display point in time,which will be described in detail below. In the case where theobservation period of time changes according to the progress of thedisplay period of time, the activity map image changes from moment tomoment along with the moving image of the monitoring area. In the casewhere the observation period of time does not change from moment tomoment according to the progress of the display period of time, theactivity map image does not change as well.

When activity information that is a source of the activity map image isgenerated, the position information may be set to be acquired fromcaptured moving images that are selected at predetermined intervals fromamong the moving images that are captured during the observation periodof time, without the need to acquire the position information from allthe moving images that are captured during the observation period oftime.

Next, the observation period of time that is to be applied when theactivity information that is the source of the activity map image isgenerated is described. FIGS. 5A, 5B, 6A, and 6B are explanatorydiagrams for describing the observation period of time that is to beapplied when the activity information that is the source of the activitymap image is generated.

According to the present embodiment, the observation period of time isset and thus the activity information is generated based on the positioninformation that is acquired from the moving image that is captured ateach point in time within the range of the observation period of time,in such a manner that the activity map image that results fromvisualizing the activity information is set to be generated. As theobservation period of time that is to be applied when the activityinformation that is the source of the activity map image is generated,there are a period-of-time movement type in which according to theprogress of the display point in time of the moving image, theobservation period of time moves following the display point in time, aperiod-of-time expansion type in which the observation period of timegradually expands, and a period-of-time-fixed type in which theobservation period of time is fixed.

FIG. 5A illustrates a case of the period-of-time movement type. In thiscase, a length T (for example, 30 minutes) of the observation period oftime is set according to the user input operation, a period of time froma period of time that is only setting length T earlier than the displaypoint in time to the display point in time is set to be the observationperiod of time, and the observation period of time moves in such amanner as to follow the display point in time according to the progressof the display point in time of the moving image. In this case, assetting information relating to the observation period of time, thesetting length of the observation period of time is set, for example,such as 30 minutes or one hour.

FIG. 5B illustrates a case of the period-of-time expansion type. In thiscase, the range of the observation period of time (a period of time froma starting point to an ending point) is set according to the user inputoperation. A starting point of a setting range of the observation periodof time is display starting point in time ts, and an ending point of thesetting range of the observation period of time is ending point in timete. In this case, as the setting information relating to the observationperiod of time, a point in time of the starting point of and a point intime of the ending point of the setting range of the observation periodof time are set, for example, in such a format as 9:00 to 12:00.

The period-of-time movement type is suitable for a case where theobservation period of time is set to be comparatively short and theactivity situation is analyzed, and the period-of-time expansion type issuitable for a case where the observation period of time is set to becomparatively long and the activity situation is analyzed. Furthermore,the period-of-time movement type is used in a case where it is desiredthat the activity situation is known during a period of time immediatelybefore the display point in time, and the period-of-time expansion typeis used in a case where, by paying attention to a specific period oftime, it is desired that the activity situation is known during such aperiod of time.

Here, according to the present embodiment, according to the user inputoperation that arbitrarily designates the display starting point in time(a reproduction starting point in time) of the monitoring moving image,processing that successively outputs the monitoring moving image at eachpoint in time starting from the designated display starting point intime is performed. At this time, in cases of the period-of-time movementtype that is illustrated in FIG. 5A and the period-of-time expansiontype that is illustrated in FIG. 5B, the observation period of time islimited to a period of time from the display starting point in time tothe display point in time.

Specifically, in the case of the period-of-time movement type that isillustrated in FIG. 5A, when an elapsed time from the display startingpoint in time to the display point in time does not agree with settinglength T of the observation period of time, a period of time before thedisplay starting point in time is excluded from the observation periodof time and the period of time from the display starting point in timeto the display point in time is an actual observation period of time. Inan example that is illustrated in FIG. 5A, in a case of display point intime t1, because an elapsed time from display starting point in time tsto display point in time t1 is shorter than setting length T of theobservation period of time, a period of time from display starting pointin time ts to display point in time t1 is the actual observation periodof time.

Therefore, in the period-of-time movement type, in a state where theelapsed time from the display starting point in time to the displaypoint in time does not agree with setting length T of the observationperiod of time, a range of the actual observation period of time isgradually widened according to the progress of the display point intime, and after the elapsed time from the display starting point in timeto the display point in time agrees with setting length T of theobservation period of time, the actual observation period of time doesnot change.

In the case of the period-of-time expansion type that is illustrated inFIG. 5B, the period of time from the display starting point in time tothe display point in time is the actual observation period of time, andwhen the display point in time reaches the ending point of the settingrange of the observation period of time, the actual observation periodof time is consistent with the setting range of the observation periodof time. In an example that is illustrated in FIG. 5B, at display pointsin time t1 and t2 in the middle of the setting range (a period of timefrom display starting point in time (a starting point) ts to displayending point in time (an ending point) te) for the observation period oftime, a period of time from display starting point in time ts to displaypoint in time t1 and display starting point in time ts to display pointt2, respectively, are the actual observation periods of time, and whenthe display point in time is the ending point of the setting range ofthe observation period of time, the entire setting range of theobservation period of time is the actual observation period of time.

Therefore, in the period-of-time expansion type, in the same manner aswith a state where, in the period-of-time movement type, the elapsedtime from the display starting point in time to the display point intime agrees with setting length T of the observation period of time, therange of the actual observation period of time is gradually widenedaccording to the progress of the display point in time, and theobservation period of time is widened within the setting range of theobservation period of time.

FIG. 6A illustrates a case where a limitation is not imposed on theobservation period of time in the period-of-time movement type.

In the example that is illustrated in FIG. 5A, in the period-of-timemovement type, the actual observation period of time is set to belimited to the period of time from the display point in time to thedisplay point in time, but in an example that is illustrated in FIG. 6A,the limitation is set not to be imposed on the observation period oftime.

Specifically, as illustrated in FIG. 6A, in a case where the limitationis not imposed on the observation period of time in the period-of-timemovement type, even if the elapsed time from the display starting pointin time to the display point in time does not agree with setting lengthL of the observation period of time, the period of time that is only thesetting length T (for example, 30 minutes) earlier than the displaypoint in time is the actual observation period of time without theperiod of time before the display starting point in time being excludedfrom the observation period of time. In the example that is illustratedin FIG. 6A, in the case of display point in time t1, the elapsed timefrom display starting point in time ts to display point in time t1 doesnot agree with setting length T of the observation period of time, but aperiod of time from point in time t1-T that is only setting length T ofthe observation period of time earlier than display point in time t1 todisplay point in time t1 is the actual observation period of time.

Furthermore, FIG. 6B illustrates a case of the period-of-time-fixed typein which the observation period of time is fixed. In this case,regardless of the display point in time, the setting range (the periodof time from display starting point in time (the starting point) ts todisplay ending point in time (the ending point) te) of the observationperiod of time is the actual observation period of time, and theactivity map image that is displayed at each point in time does notchange. The period-of-time-fixed type is the same as the period-of-timeexpansion type that is illustrated in FIG. 5B, in that the range of theobservation period of time is set according to the user input operation,and the period-of-time expansion type can be considered as a type thatresults from limiting the actual observation period of time to theperiod of time from the display starting point in time to the displaypoint in time in the period-of-time-fixed type.

Here, because, in methods that are illustrated in FIGS. 5A, 5B, and 6A,a period of time that is later than the display point in time is notincluded in the observation period of time, these methods can be appliedto both a real-time moving image display mode in which a monitoringmoving image is generated in real time from a current moving imagecaptured by camera 1 and thus the generated monitoring moving image isdisplayed, and a stored-moving-image display mode in which a monitoringmoving image is generated from past moving images that are stored inrecorder 2 and thus the generated monitoring moving image is displayed.In contrast, because, in a method that is illustrated in FIG. 6B, theperiod of time that is later than the display point in time is includedin the observation period of time, this method can be applied only tothe stored-moving-image display mode.

Furthermore, in the methods that are illustrated in FIGS. 5A and 5B,information at a point in time earlier than the display starting pointin time and information at a point in time later than the display pointin time are not included in the activity map image. In contrast, in themethod that is illustrated in FIG. 6A the information at a point in timeearlier than the display starting point in time is included in theactivity map image, and in the method that is illustrated in FIG. 6B,the information at a point in time later than the display point in timeis included in the activity map image. For this reason, in methods thatare illustrated in FIGS. 6A and 6B, there is a case where informationthat, judging from a purpose of activity situation analysis, gets in theway of the activity situation analysis is included in the activity mapimage. In such a case, as illustrated in FIGS. 5A and 5B, a method ofimposing the limitation on the observation period of time may beemployed. For example, in a case where it is desired that the activitysituation is analyzed during a time span such as an afternoon, thedisplaying of the moving image starts at 13:00, but there is a casewhere it is not desired that information at a point in time earlier than13:00 is included in the activity map image. In such a case, asillustrated in FIGS. 5A and 5B, the method of imposing the limitation onthe observation period of time may be selected.

According to the present embodiment, as illustrated in FIG. 6B, in thecase of the period-of-time-fixed type, one observation period of time isset, but the observation period of time may be set successively, forexample, at every time span, in such a manner that the activity mapimage is generated and displayed. In this case, at a predeterminedinterval corresponding to the observation period of time, the activitymap image is periodically updated.

Next, processing that generates the activity information that is thesource of the activity map image is described. FIG. 7 is an explanatorydiagram for describing the processing that generates the activityinformation that is the source of the activity map image. FIG. 7illustrates an example of the period-of-time movement type that isillustrated in FIGS. 5A and 6A, but the same processing is performedeven in cases of the period-of-time expansion type that is illustratedin FIG. 5B and the period-of-time-fixed type that is illustrated in FIG.6B.

According to the present embodiment, the position information on everyperson at each point in time is acquired from the moving image (frame)that is captured at each point in time, the temporal statisticalprocessing is performed on the position information on every person ateach point in time, and thus the pieces of activity information (thestaying time and the number of stays) relating to the activity situationof the person are acquired. However, at this time, first, the activityinformation during every unit time (for example, 10 seconds, 30 seconds,one minute, or the like) that is set to be sufficiently shorter than theobservation period of time (for example, 30 minutes, one hour, or thelike), is generated, the activity information during every unit time isaggregated during the observation period of time, and thus the activityinformation is acquired during the observation period of time.

Here, a method of calculating the number of stays during the observationperiod of time is described. In this case, the number of motion lines ofeach person who passes through the observation area during the unit timeis counted, and thus, the number of stays (the number of stayingpersons) during every unit time is obtained. Processing (statisticalprocessing) that simply adds the number of stays during every unit timeis performed, and thus the number of stays during the observation periodof time can be obtained.

Next, a method of calculating the staying time during the observationperiod of time is described. In this case, for the motion line of eachperson that passes through the observation area during the unit hour,the staying time for which every person stays during the staying time (apoint in time at which the person enters the observation area from apoint in time at which the person leaves the observation area) isacquired, the staying time for which every person stays is nextcalculated, averaging processing (statistical processing) is thenperformed on the staying time for which every person stays, and thus thestaying time during every unit time is acquired. The averagingprocessing (the statistical processing) is performed on the staying timeduring every unit time, and thus the staying time during the observationperiod of time can be obtained.

Furthermore, as illustrated in FIG. 5A, in the case of theperiod-of-time movement type, the observation period of time moves insuch a manner as to follow the display point in time according to theprogress of the display point in time of the moving image. However, asillustrated in FIG. 7, in a method of acquiring the activity informationduring every unit time, a state where the observation period of time isshifted by the unit time each time the display point in time progressesby the unit time is attained, and according to this, the activityinformation during the observation period of time is updated. At thistime, processing is only performed that excludes and integrates theactivation information during the unit time that is a difference betweena current observation period of time and a previous observation periodof time, from and into the activity information during the previousobservation period of time, and thus the activity information during thecurrent observation period of time can be acquired. That is, during theunit time within the previous observation period of time, if theactivity information during a unit time other than the currentobservation period of time is excluded from the activity informationduring the previous observation period of time, and at the same time theactivity information during the unit time that is newly added to thecurrent observation period of time is integrated into the activityinformation during the previous observation period of time, the activityinformation during the current observation period of time can beacquired.

Furthermore, in a state where the elapsed time from the display startingpoint in time to the display point in time does not agree with settinglength T of the observation period of time, because only the endingpoint of the observation period of time moves according to the progressof the display point in time, if the activity information during theunit time that is newly added to the current observation period of timeis integrated into the activity information during the previousobservation period of time, the activity information during the currentobservation period of time can be acquired.

If the display point in time progresses within a unit time, there is nochange during the observation period of time, and when the display pointin time reaches the ending point of the unit time, processing thatupdates the activity information during the observation period of timeis performed. For this reason, if the display point in time progresseswithin the range of the unit time, there is no change in the activitymap image, and when the display point in time proceeds to the next unittime, the activity map image is updated. That is, each time when thedisplay point in time progresses by the unit time, the activity mapimage is updated.

On the other hand, as illustrated in FIG. 5B, in the case of theperiod-of-time expansion type, because only the ending point of theobservation period of time moves according to the progress of thedisplay point in time, if the activity information during the unit timethat is newly added to the current observation period of time isintegrated into the activity information during the observation periodof time, which is acquired last time, the activity information duringthe current observation period of time can be acquired.

Next, processing that is performed on PC 3 that is illustrated in FIG. 1is described. FIG. 8 is a functional block diagram illustrating aschematic configuration of PC 3.

PC 3 includes moving image acquirer 31, position information acquirer32, position information storage 33, activity information acquirer 34,activity information storage 35, observation-period-of-time controller36, activity information aggregator 37, activity map image generator 38,output controller 39, processing condition setter (an observationcondition setter and a unit time setter) 40.

In moving image acquirer 31, a moving image of the inside of the store,which is captured by camera 1 is acquired from camera 1 or recorder 2.Here, in a case where a current moving image is displayed in real time,a moving image is acquired from camera 1, and in a case where a pastmoving image is displayed, a moving image is acquired from recorder 2.

Position information acquirer 32 detects a person from the moving imagethat is acquired in moving image acquirer 31 and acquires the positioninformation on every person, and includes person detector 45 and motionline information acquirer 46.

In person detector 45, processing that detects a person from the movingimage (a frame) acquired in moving image acquirer 31 is performed. Theprocessing that is performed in person detector 45 may use a known imagerecognition technology. In motion line information acquirer 46, based ona result of the detection in person detector 45, processing thatacquires a motion line of every person is performed. The pieces ofinformation relating to the motion line of every person, which areacquired in motion line information acquirer 46, are stored in positioninformation storage 33 as position information on every person.

In position information acquirer 32, coordinates of a person's centralpoint are acquired, and the motion line is generated in such a mannerthat the person's central points are connected. Furthermore, timeinformation relating to a point in time at which every person isdetected and the like, which are acquired starting from a point in timeat which a moving image from which a person is detected is captured isincluded in the position information that is acquired in positioninformation acquirer 32.

Activity information acquirer 34 performs the temporal statisticalprocessing on position information (motion line information) of a personthat is acquired by position information acquirer 32 and is stored inposition information storage 33, and acquires the pieces of activityinformation (the staying time and the number of stays) relating to theactivity situation of the person in each observation area (refer to FIG.3A) per unit time. Activity information acquirer 34 includesnumber-of-stays acquirer 47 and staying time acquirer 48. The activityinformation during every unit time, which is acquired in activityinformation acquirer 34 is stored in activity information storage 35.

In number-of-stays acquirer 47, based on motion line information onevery person that is stored in position information storage 33,processing that acquires the number of stays (the number of stayingpersons) during every unit time is performed. In this processing, thenumber of motion lines of each person who passes through the observationarea during the unit time is counted, and thus, the number of staysduring every unit time is obtained. This processing is performed onevery observation area.

In staying time acquirer 48, based on the motion line information onevery person that is stored in position information storage 33, theprocessing that acquires the staying time during every unit time isperformed. In this processing, first, for the motion line of each personthat passes through the observation area during the unit hour, thestaying time for which every person stays during the staying time (apoint in time at which the person enters the observation area from apoint in time at which the person leaves the observation area) isacquired, the staying time for which every person stays is nextacquired, averaging processing (statistical processing) is thenperformed on the staying time for which every person stays, and thus thestaying time during every unit time is acquired. This processing isperformed for every observation area.

In activity information aggregator 37, processing is performed that,during the observation period of time, aggregates the pieces of activityinformation (the staying time and the number of stays) during the unittime that is acquired in activity information acquirer 34, and thatacquires the activity information during the observation period of time.In this processing, the processing (the statistical processing) thatsimply adds the number of stays during every unit time is performed andthe activity information during the observation period of time isobtained. Furthermore, the averaging processing (the statisticalprocessing) is performed on the staying time during the unit time andthe staying time during the observation period of time is acquired.These processing operations are performed for every observation area.

In observation-period-of-time controller 36, control of the observationperiod of time is performed in accordance with conditions of theobservation period of time that are set in processing condition setter40. According to the present embodiment, as the conditions relating tothe observation period of time, an observation period-of-time type (theperiod-of-time movement type, the period-of-time expansion type, or theperiod-of-time-fixed type), the presence or absence of the limitation ofthe observation period-of-time during the period-of-time movement type,and the length and the range of the observation period of time are setin processing condition setter 40, and, based on the conditions of theobservation period of time, and the display starting point in time ofand the current display point in time of the monitoring moving image,the control of the observation period of time is performed inobservation-period-of-time controller 36. Based on the observationperiod of time that is controlled by observation-period-of-timecontroller 36, activity information aggregation processing is performedin activity information aggregator 37.

In activity map image generator 38, processing is performed thatgenerates the activity map image which results from visualizing thepieces of activity information (the number of stays and the stayingtime) during the observation period of time, which is acquired inactivity information aggregator 37. If a display form (a color ortransmissivity) of the activity map image is set in advance, this may besufficient, but the display form may be set to be able to be suitablyselected by the user.

In the output controller 39, processing is performed that generates themonitoring image which results from superimposing the activity map imagethat is generated in activity map image generator 38, onto the movingimage of the monitoring area that is acquired in moving image acquirer31, at every predetermined point in time. Accordingly, the monitoringscreen (refer to FIG. 9) is displayed on monitor 7. Furthermore, inoutput controller 39, processing is performed that displays a processingcondition setting screen (refer to FIG. 14) on monitor 7.

Furthermore, in output controller 39, processing is performed thatcontrols an output of the monitoring moving image according to the userinput operation. Particularly, in output controller 39, processing isperformed that, according to the user's input operation that arbitrarilydesignates the display starting point in time of the monitoring movingimage, generates the monitoring moving image from the designated displaystarting point in time and outputs the generated monitoring movingimage.

In processing condition setter 40, according to the user's inputoperation, processing is performed that sets various conditions that areto be met when generating the activity map image. According to thepresent embodiment, processing that sets a position of the observationarea is performed according to the user's input operation. Here, insetting the position of the observation area, if an input screen onwhich the image of the inside of the store is displayed is displayed onmonitor 7, a position of the target area is input by the user throughthe input screen, and thus position information on the target area isacquired, this may be sufficient. In accordance with the positioninformation on the target area that is set in processing conditionsetter 40, processing that acquires the position information on everyperson is performed in position information acquirer 32.

Furthermore, in processing condition setter 40, according to the userinput operation, processing is performed that sets the observationperiod-of-time type (the period-of-time movement type, theperiod-of-time expansion type, or the period-of-time-fixed type), thepresence or absence of the limitation of the observation period-of-timeduring the period-of-time movement type, and the length and the range ofthe observation period of time, as the conditions relating to theobservation period of time. Furthermore, in processing condition setter40, according to the user input operation, processing that sets the unittime is performed.

In setting the condition relating to the observation period of time andthe unit time, the processing condition setting screen (refer to FIG.14) may be displayed on monitor 7 and an input operation designating thecondition relating to the observation period of time and the unit timemay be performed by the user on the processing condition setting screen,in such a manner that information on the condition relating to theobservation period of time and on the unit time is acquired. Inaccordance with the conditions relating to the observation period oftime and the unit time that are set in processing condition setter 40,predetermined processing operations are performed in activityinformation acquirer 34, observation-period-of-time controller 36, andactivity information aggregator 37.

Furthermore, according to the present embodiment, input relating to oneor more processing conditions is performed on the monitoring screen(refer to FIG. 9) on which the monitoring moving image is displayed, andthe monitoring screen serves as the processing condition setting screentoo. Particularly, the processing conditions that are desired to be setin advance are designated by the user on the processing conditionsetting screen (refer to FIG. 14), and the processing conditions thatcan be set whenever necessary when the monitoring moving image isbrowsed through are designed by the user on the monitoring screen.

PC 3 that is illustrated in FIG. 8 has a processor (a CPU) and a programmemory. Each unit of PC 3 is realized by executing an applicationprogram (instructions) for the activity situation analysis in the CPU ofPC 3. This program may be not only configured as a dedicated device thatis introduced in advance in PC 3 as the information processingapparatus, but may be also set to be provided to the user in a state ofbeing recorded on a suitable program recording medium, as an applicationprogram running on a general-purpose OS, or through a network.

Next, the monitoring screen that is displayed on monitor 7 which isillustrated in FIG. 8 is described. FIG. 9 is an explanatory diagramillustrating the monitoring screen that is displayed on monitor 7. FIGS.10A and 10B are explanatory diagrams illustrating display point-in-timeand observation period-of-time designation section 59 that isillustrated in FIG. 9.

The user browses through this monitoring screen in order to know theactivity situation of the customers within the store. Included in themonitoring screen are store selection section 51, starting button 52,setting button 53, operation mode selection section 54, date and timedesignation section 55, moving image operation section 56, moving imagedisplay mode selection section 57, observation period-of-time typeselection section 58, display point-in-time and observationperiod-of-time designation section 59, scale regulator 60, and movingimage display section 61.

In store selection section 51, the user selects a store that is amonitoring target, from among multiple stores. Here, the store is set tobe selected with a pull-down menu. With starting button 52, activitysituation analysis processing is started on PC 3. With setting button53, various conditions in the activity situation analysis processing areset. According to the present embodiment, when setting button 53 isoperated, processing condition setting screen (refer to FIG. 14) isdisplayed in a pop-up manner. In operation mode selection section 54, anoperation mode is selected. Operation modes, such as salesroommonitoring, commodity monitoring, display alert, and stockout alert, areprepared. The monitoring screen that is illustrated in FIG. 9 isdisplayed by selecting the salesroom monitoring.

The monitoring moving image generated in the output controller 39, thatis, the monitoring image (refer to FIG. 3B) that results fromsuperimposing the activity map image onto the moving image of themonitoring area is displayed in a motion video format on moving imagedisplay section 61. Particularly, in an example that is illustrated inFIG. 9, two images that result from panoramically expanding a movingimage which is captured by camera 1 that is an omnidirectional cameraare displayed. The entire store is shown in the two images, and asituation of the entire store can be checked.

In date and time designation section 55, the user designates a date andtime (year, month, date, hour, minute and the like). Here, the date andtime is set to be directly input, but a calendar screen may be displayedin such a manner that the date and time may be set to be able todesignated. When the date and time is input with date and timedesignation section 55, the monitoring moving image of the data and timethat is input is displayed on moving image display section 61. When areal-time moving image display mode is selected in moving image displaymode selection section 57, a current date and time is displayed.

In moving image display mode selection section 57, the user selects themoving image display mode (the real-time moving image display mode orthe stored-moving-image display mode). Here, when the real-time movingimage display mode is selected, the monitoring moving image on which acurrent situation is displayed in real time is displayed on moving imagedisplay section 61, and when the stored-moving-image display mode isselected, the monitoring moving image on which a situation at a date andtime that is designated in date and time designation section 55 isdisplayed is displayed on moving image display section 61.

In observation period-of-time type selection section 58, the userselects the observation period-of-time type (the period-of-time movementtype, the period-of-time expansion type, or the period-of-time-fixedtype). Here, when the period-of-time movement type is selected, based onthe length of the observation period of time that is designated indisplay point-in-time and observation period-of-time designation section59, as illustrated in FIGS. 5A and 6A, control is performed thatgradually moves the observation period of time according to the progressof the display point in time. When the period-of-time expansion type isselected, based on the range of the observation period of time that isdesignated in display point-in-time and observation period-of-timedesignation section 59, as illustrated on FIG. 5B, control is performedthat gradually widens an actual observation period of time according tothe progress of the display point in time within the setting range ofthe observation period of time. When the period-of-time-fixed type isselected, as illustrated in FIG. 6B, control is performed that fixes theobservation period of time.

In display point-in-time and observation period-of-time designationsection 59, the user designates the display point in time of the movingimage of the monitoring area, and the length or the range of theobservation period of time, which are displayed on moving image displaysection 61. Displayed on display point-in-time and observationperiod-of-time designation section 59 are scale 63 indicating a timeaxis, slider (a display point-in-time operation section) 64 indicatingthe display point in time of the moving image, slider (an observationperiod-of-time operation section) 65 a indicating a starting point ofthe observation period of time, and observation period-of-time displaysection 66 indicating the range of the observation period-of-time thatis designated by sliders 64 and 65 a.

Slider 64 indicates a current point in time in the real-time movingimage display mode, and indicates a reproduction point in time in thestored-moving-image display mode. Furthermore, in thestored-moving-image display mode, an operation (a click or a drag) thatmoves slider 64 along scale 63 using input device 6 such as a mouse canbe performed. Slider 64 is operated, and thus the monitoring movingimage at a desired point in time can be displayed on moving imagedisplay section 61.

Furthermore, FIG. 9 illustrates the case of the period-of-time movementtype, and slide 65 a indicates the starting point of the observationperiod of time. FIGS. 10A and 10B illustrate the cases of theperiod-of-time expansion type and the period-of-time-fixed type,respectively. Sliders (the observation period-of-time operationsections) 65 a and 65 b indicate the starting point and the ending pointof the observation period of time, respectively. In the case of theperiod-of-time movement type that is illustrated in FIG. 9, because thedisplay point in time of the moving image of the monitoring area and theending point of the observation period of time are consistent with eachother, slider 65 b does not indicate the ending point of the observationperiod of time.

In the stored-moving-image display mode, as is the case with slider 64,an operation (a click or a drag) that moves sliders 65 a and 65 b alongscale 63 can be performed using input device 6. In the case of theperiod-of-time movement type that is illustrated in FIG. 9, slider 65 ais operated, and thus the length of the observation period of time canbe designated. In the cases of the period-of-time expansion type and theperiod-of-time-fixed type that are illustrated in FIGS. 10A and 10B,respectively, sliders 65 a and 65 b are operated, and thus the range ofthe observation period of time can be designated.

In scale regulator 60, the user designates a scale interval of scale 63of display point-in-time and observation period-of-time designationsection 59. Here, the scale interval (for example, fifteen minutes,thirty minutes, one hour, one day, or the like) is set to be selectedwith the pull-down menu. In examples that are illustrated in FIGS. 9 and10A, the scale-mark interval is set to thirty minutes, and in an examplethat is illustrated in FIG. 10B, the scale interval is set to be oneday. Furthermore, in the display point-in-time and observationperiod-of-time designation section 59, a display range of scale 63 ischanged according to the scale interval that is designated in scaleregulator 60. In the examples that are illustrated in FIGS. 9 and 10A, aperiod of time from 10:00 to 14:00 is displayed, and in the example thatis illustrated in FIG. 10B, a period of time from a date of 24 to a dateof 1 of the next month is displayed.

When the scale interval that is designated in scale regulator 60 may beset to be a minimum unit that is used when the display point in time orthe observation period of time is designated. That is, the display pointin time or the observation period of time may be set to be able to beadjusted with the scale interval. For example, when the scale intervalis set to be 30 minutes, the display point in time or the observationperiod of time can be adjusted at intervals of 30 minutes. Furthermore,the unit time may be set to be displayed on scale 63 of displaypoint-in-time and observation period-of-time designation section 59. Inthis case, the scale interval that is designated in scale regulator 60may be displayed as a large scale, and the unit time may be displayed asa small scale.

In moving image operation section 56, an operation relating toreproduction of the monitoring moving image that is displayed on movingimage display section 61 is performed. Operation buttons for normalreproduction, fast forwarding, rewinding, and stop are provided inmoving image operation unit 56. The monitoring moving image can beefficiently browsed through over a long period of time by operatingthese operation buttons. Particularly, in the stored-moving-imagedisplay mode, slider 64 of display point-in-time and observationperiod-of-time designation section 59 is operated to designate areproduction starting position, moving image operation section 56 isoperated, and thus the monitoring moving image can be browsed throughstarting from a desired point in time.

Next, a different example of display point-in-time and observationperiod-of-time designation section 59 that is illustrated in FIG. 9 isdescribed. FIG. 11 is an explanatory diagram for describing thedifferent example of display point-in-time and observationperiod-of-time designation section 59.

In the example that is illustrated in FIG. 9, display point-in-time andobservation period-of-time designation section 59 is provided in themonitoring screen, and designation of the display point in time and theobservation period of time is performed in scale 63, but in an examplethat is illustrated in FIG. 11, display point-in-time and observationperiod-of-time designation section 59 is set to be provided in a stateof being divided into display point-in-time designation section 71 andobservation period-of-time designation section 72 in such a manner thatthe designation of the display point in time and the observation periodof time can be performed with different scales 63.

In display point-in-time designation section 71, the user designate thedisplay point in time. With observation period-of-time designationsection 72, the user designates the observation period of time. Twoscale regulators 60 are correspondingly provided in displaypoint-in-time designation section 71 and observation period-of-timedesignation section 72, respectively. The scale intervals can bedesignated for display point-in-time designation section 71 andobservation period-of-time designation section 72, respectively. In anexample that is illustrated in FIG. 11, in display point-in-timedesignation section 71, the scale interval is set to be 30 minutes, andin observation period-of-time designation section 72, the scale intervalis set to be one day.

Accordingly, the designation of the display point in time and theobservation period of time can easily be performed. That is, in a casewhere the observation period of time is designated with its unit as oneday, the observation period of time may be designated with the scaleinterval of scale 63 as one day, but in this scale 63, the display pointin time (the reproduction point in time) is difficult to designate.Conversely, when the scale interval of scale 63 is set to be 30 minutesin such a manner that the display point in time is easy to designate,the observation period of time is difficult to designate with its unitas one day. In this case, when the designation of the display point intime and the observation period of time is set to be able to beperformed with different scales 63, the display point in time and theobservation period of time are easy to designate.

Next, a transition situation of display point-in-time and observationperiod-of-time designation section 59 that is illustrated in FIG. 9 isdescribed. FIGS. 12A and 12B are explanatory diagrams illustrating thetransition situation of display point-in-time and observationperiod-of-time designation section 59.

FIG. 12A illustrates a case of the period-of-time movement type. In thiscase, according to the progress of the display point in time, slider 64indicating the display point in time moves. At the same time, slider 65a indicating starting point of the observation period of time andobservation period-of-time display section 66 indicating the range ofthe observation period of time are integrated into one piece and movesfollowing slider 64 indicating the display point in time.

FIG. 12B illustrates the cases of the period-of-time expansion type andthe period-of-time-fixed type. In these cases, slider 64 indicating thedisplay point in time moves according to the progress of the displaypoint in time, but sliders 65 a and 65 b indicating the starting pointand the ending point of the observation period of time, respectively,and observation period-of-time display section 66 indicating the rangeof the observation period of time remain motionless.

Next, different examples of the display point in time and theobservation period of time are described.

FIGS. 13A and 13B are explanatory diagrams illustrating the transitionsituation of display point-in-time and observation period-of-timedesignation section 59 in the different examples of the display point intime and the observation period of time.

In the example of the period-of-time movement type that is illustratedin FIG. 12A, the period of time that is only the setting length of theobservation period of time earlier than the display point in time is setto be the observation period of time, and the display point in time isthe ending point of the observation period of time. However, asillustrated in FIG. 13A, the display point in time and the observationperiod of time may be in a state of being at a distance from each other,that is, the period of time that is only the setting length earlier thanthe point in time that is a predetermined time earlier than the displaypoint in time may be the observation period of time, in such a mannerthat, according to the progress of the display point in time, theobservation period of time moves at a predetermined distance from thedisplay point in time. In this case, in a state of being at a distancefrom slider 64 indicating the display point in time of the moving image,sliders 65 a and 65 b indicating the observation period of time and theobservation period-of-time display section 66 move in such a manner asto follow the display point in time.

Furthermore, in the examples of the period-of-time expansion type andthe period-of-time-fixed type that are illustrated in FIG. 12B, thedisplay point in time is within the range of the observation period oftime, but as illustrated in FIG. 13B, the display starting point in timemay be designated out of the range of the observation period of time,that is, the display of the moving image may be started from a positionthat is at a distance from the observation period of time. In this case,slider 64 indicating display point in time starts to move from theposition that is at a distance from the observation period of time,sliders 65 a and 65 b indicating the starting point and the ending pointof the observation period of time, respectively, and observationperiod-of-time display section 66 remain motionless.

Particularly, in a method that is illustrated in FIG. 13B, for example,a usage method is possible in which an afternoon moving image is viewedwhile a morning activity map image is displayed, that is, theobservation period of time is designated as a morning time span (from10:00 to 12:00) and the display starting point in time of the movingimage is designated as an afternoon point in time (13:00). Furthermore,there is a usage method in which a today moving image is viewed while ayesterday activity map image at the same time span is displayed.

In the case of the period-of-time movement type, it is possible todesignate the display point in time in such a manner that the displaypoint in time is positioned at a predetermined position within theobservation period of time. In this case, according to the progress ofthe display point in time, the observation period of time moves with thedisplay point in time in a state of covering before and after thedisplay point in time.

Next, processing that is performed in processing condition setter 40which is illustrated in FIG. 8 is described. FIG. 14 is an explanatorydiagram illustrating the processing condition setting screen that isdisplayed on monitor 7.

According to the present embodiment, in processing condition setter 40,according to the user input operation, the processing is performed thatsets various conditions that are to be met when generating the activitymap image. At this time, the processing condition setting screen that isillustrated in FIG. 14 is displayed on monitor 7. In the processingcondition setting screen, the user designates various conditions thatare to be met when generating the activity map image, and the processingcondition setting screen is displayed in a pop-up manner according tothe operating of setting button 53 on the monitoring moving image (referto FIG. 9).

Included in the processing condition setting screen are area settingbutton 81, unit time designation section 82, observation period-of-timelimitation designation section 83, and setting button 84.

When area setting button 81 is operated, the input screen on which themoving image of the inside of the store is displayed is displayed, andthe user performs an operation of inputting a position of theobservation area into the input screen using input device 6 such as amouse. Without being into a different screen, the observation area maybe set to be able to be input into the processing condition settingscreen.

In unit time designation section 82, the user designates a length of theunit time. Here, the length (for example, 10 seconds, 30 seconds, oneminute, or the like) of the unit time may be selected from the pull-downmenu. Based on the unit time that is designated in unit time designationsection 82, as illustrated in FIG. 7, processing is performed thatgenerates the activity information during every unit time.

The unit time is a minimum unit of the observation period of time, andthe observation period of time can be adjusted to a length of the unittime. For this reason, the unit time is changed, and thus the extent towhich the observation period of time is adjusted can be changed. Forexample, in a case where the extent to which the observation period oftime is adjusted is desired to be small, the unit time may be shortened.Furthermore, normally, because the changing of the unit time isinfrequently performed, the unit time may be set to be a predeterminedtime (for example, 30 seconds), and processing condition setting may beomitted.

In observation period-of-time limitation designation section 83, theuser designates the presence or absence of limitation of the observationperiod of time, that is, whether or not the observation period of timeis limited to the period of time from the display starting point in timeto the display point in time. Here, when “limiting” is selected, asillustrated in FIGS. 5A and 5B, processing is performed that limits theobservation period of time. When “do not limit” is selected, asillustrated in FIGS. 6A and 6B, processing is performed that does notlimit the observation period of time.

As described above, the user performs necessary operations in areasetting button 81, unit time designation section 82, and observationperiod-of-time limitation designation section 83, and when settingbutton 84 is operated, in processing condition setter 40, processingoperations are performed that sets matters relating to the observationarea, the unit time, and the presence or absence of limitation of theobservation period of time in accordance with contents that aredesignated by the user.

Next, a different example of the processing condition setting screenthat is illustrated in FIG. 14 is described.

FIG. 15 is an explanatory diagram illustrating the different example ofthe processing condition setting screen that are illustrated in FIG. 14.

In the example that is illustrated in FIG. 9, on the monitoring screen,the type of observation period of time, and the length and the range ofthe observation period of time are set to be designated, but in theexamples that are illustrated in FIGS. 12A and 12B, on the processingcondition setting screen, the type of observation period of time, andthe length and the range of the observation period of time are set to bedesignated. In addition to area setting button 81 that is provided inthe processing condition setting screen that is illustrated in FIG. 14,unit time designation section 82, observation period-of-time limitationdesignation section 83, and setting button 84, observationperiod-of-time type selection section 85, observation period-of-timelength designation section 86, scale regulator 87, and observationperiod-of-time range designation section 88 are included in processingcondition setting screen.

In observation period-of-time type selection section 85, in the samemanner as in observation period-of-time type selection section 58 thatis illustrated in FIG. 9, the user selects the type of observationperiod of time (the period-of-time movement type, the period-of-timeexpansion type, or the period-of-time-fixed type).

In observation period-of-time length designation section 86, the userdesignates the length of the observation period of time in theperiod-of-time movement type. Here, the length of the observation periodof time is set to be selected from the pull-down menu. The length of theobservation period of time may be set to be able to be input as anumerical value.

In observation period-of-time range designation section 88, the userdesignates the range of the observation period of time in theperiod-of-time expansion type and the period-of-type-fixed type. Here,in the same manner as in the example that is illustrated in FIG. 9, theuser designates the range of the observation period of time with sliders65 a and 65 b, and the range of the observation period of time that isspecified by sliders 65 a and 65 b is displayed on observationperiod-of-time display section 66. In scale regulator 87, the userdesignates the scale interval of scale 63 of observation period-of-timerange designation section 88.

According to the present embodiment described above, the followingprocessing operations are set to be performed. In position informationacquirer 32, a person is detected from the moving image of themonitoring area and thus the position information on every person isacquired. In activity information acquirer 34, the activity informationis acquired during every unit time based on the position information onevery person. In processing condition setter (the observation conditionsetter) 40, the conditions relating to the observation period of timefor observing the activity situation of the person are set according tothe user input operation. In observation-period-of-time controller 36,the observation period of time is controlled in accordance with theconditions relating to the observation period of time that is set byprocessing condition setter 40. In activity information aggregator 37,the activity information during every unit time is aggregated based onthe observation period of time that is controlled byobservation-period-of-time controller 36 and the activity information isacquired during the observation period of time is acquired. In activitymap image generator 38, the activity map image is generated based on theactivity information during the observation period of time. In outputcontroller 39, the monitoring moving image including the activity mapimage and the moving image of the monitoring area is generated at apredetermined point in time and the generated monitoring moving image isoutput to the display device. For this reason, because the monitoringmoving image including the activity map image and the moving image ofthe monitoring area is generated at a predetermined point in time andthe generated monitoring moving image is output, while the moving imageof the monitoring area is in the middle of being displayed, the user caneasily know how the activation situation of the person changes.Furthermore, because the observation period of time is controlled inaccordance with the conditions that are set according to the user inputoperation, the customization that variously changes the analysis form ofthe activity situation according to the user need can be easilyperformed.

Furthermore, according to the present embodiment, in output controller39, because the monitoring moving image that results from superimposingthe activity map moving image onto the moving image of the monitoringarea is generated and is output, the user can immediately know theactual situation of the person in the attention-receiving area withinthe monitoring area and the activity situation of the person.

Furthermore, according to the present embodiment, inobservation-period-of-time controller 36, the observation period of timeis set to be controlled in such a manner that, according to the progressof the display point in time of the moving image of the monitoring area,the observation period of time moves following the display point intime. For this reason, with the activity map image, the user can knowthe activity situation of the person during a period of time that isbased on the display point in time of the moving image of the monitoringarea. When the display point in time progresses by the unit time, theobservation period of time is moved by the unit time, and the activityinformation during the observation period of time is updated, but atthis time, processing is only performed that excludes and integrates theactivity information during the unit time, which is equivalent to adifference between a new observation period of time and the previousobservation period of time, from and into the activity informationduring the previous observation period of time, and thus the activityinformation during the new observation period of time can be acquired.Because of this, the arithmetic operation load can be reduced.

Furthermore, according to the present embodiment, inobservation-period-of-time controller 36, the observation period of timeis set to be controlled in such a manner that, based on the settinglength of the observation period of time that is set by processingcondition setter (the observation condition setter) 40, the period oftime that is by the setting length of the observation period of timeearlier than the display point in time of the moving image of themonitoring area is the observation period of time. Because of this, withthe activity map image, the user can know the activity situation of theperson during the period of time immediately before the moving image ofthe monitoring area.

Furthermore, according to the present embodiment, inobservation-period-of-time controller 36, until the elapsed period oftime from the display starting point in time of the moving image of themonitoring area to the display point in time agrees with the settinglength of the observation period of time, the observation period of timeis set to be controlled in such a manner that the period of time fromthe display starting point in time to the display point in time is theobservation period of time, and the information that is earlier than thedisplay starting point in time of the moving image of the monitoringarea is not included in the activity map image. Because of this, theuser can suitably know the activity situation of the person during theperiod of time at an attention-receiving point in time or later.

Furthermore, according to the present embodiment, inobservation-period-of-time controller 36, the observation period of timeis set to be controlled in such a manner that, based on the settingrange of the observation period of time, which is set by processingcondition setter (the observation condition setter) 40, within thesetting range of the observation period of time, the period of time fromthe display starting point in time of the moving image of the monitoringarea, which is the starting point of the observation period of time, tothe display point in time is the observation period of time. For thisreason, with the activity map image, the user can know the activitysituation of the person that occurs after displaying of the moving imageof the monitoring area is started. When the display point in timeprogresses by the unit time, the observation period of time is moved bythe unit time, and the activity information during the observationperiod of time is updated, but at this time, the activity informationduring the unit time, which is equivalent to a difference between theprevious observation period of time and a new observation period oftime, is only integrated into the activity information during theprevious observation period of time, and thus the activity informationduring the new observation period of time can be acquired. Because ofthis, the arithmetic operation load can be reduced.

Furthermore, according to the present embodiment, in processingcondition setter (the observation condition setter) 40, the length orthe range of the observation period of time is set according to the userinput operation that arbitrarily designates the length or the range ofthe observation period of time as the conditions of the observationperiod of time. Because the user can arbitrarily designate the length orthe range of the observation period of time, the customization attributecan be improved that changes the analysis form of the activitysituation.

Furthermore, according to the present embodiment, in output controller39, because the monitoring moving image including the activity map imageand the current moving image of the monitoring image is generated inreal time and is output, the user can browse through the activity mapimage and the current moving image of the monitoring area in real time.

Furthermore, according to the present embodiment, in processingcondition setter (the unit time setter) 40, the unit time is setaccording to the user input operation, and in activity informationacquirer 34, the activity information during every unit time is set tobe acquired based on the unit time that is set by processing conditionsetter 40. Thus, because the user can designate the unit time, thecustomization attribute can be improved that changes the analysis formof the activity situation.

Second Embodiment

Next, a second embodiment is described. What is not described here isthe same as in the first embodiment. FIGS. 16A and 16B are explanatorydiagrams illustrating the activity map image that is an area divisiontype according to the second embodiment.

According to the first embodiment, as illustrated in FIGS. 3A and 3B,the observation area is set to be within the monitoring area (the movingimage of which is captured by camera 1), and the activity map image thatresults from visualizing the pieces of activity information (the stayingtime and the number of stays) on every observation area is set to bedisplayed in a state of being superimposed onto the moving image of themonitoring area. However, according to the second embodiment, asillustrated in FIGS. 16A and 16B, all the monitoring areas are set to bethe observation areas, the observation areas are divided into multiplegrids (division areas), and the activity map image indicating theactivity information on every grid is set to be displayed in the stateof being superimposed on the moving image of the monitoring area. Withthe activity map image, a distribution situation of the activityinformation in the monitoring area can be known.

Furthermore, in the activity map image that is the area division type,at least one among the display elements that are the color, theintensity, and the pattern (the design), and the transmissivity ischanged for every grid, and thus a numerical value of the activityinformation on every grid can be expressed. In examples that areillustrated in FIGS. 16A and 16B, the number of stays in every grid isexpressed with color intensity of the grid (intensity of filling color).

In the activity map images that are illustrated in FIGS. 3A and 3B, thetwo pieces of activity information, that is, the staying time and thenumber of stays are visualized with the color intensity of and the sizeof the activity map image at the same time. On the other hand, in theactivity map images that are illustrated in FIGS. 16A and 16B, one pieceof activity information, that is, only the number of stays, isvisualized, but the activity map image that results from visualizing thestaying time instead of the number of stays may be available.Furthermore, the activity map image that results from visualizing eachof the staying time and the number of stays may be generated and may bedisplayed as a different monitoring moving image.

Furthermore, in order to generate the activity map image indicating thenumber of stays, the number of motion lines of a person that passes eachgrid during the observation period of time may be counted and the numberof stays in every grid may be obtained. Furthermore, if the staying timefor which every person stays during the target period of time isacquired in every grid, the staying time and the number of stayingpersons in every grid can be acquired at the same time. Accordingly, theactivity map image indicating the number of stays and the activity mapimage indicating the staying time can be generated at the same time.

Third Embodiment

Next, a third embodiment is described. What is not described here is thesame as in the first embodiment. FIG. 17 is a functional block diagramillustrating schematic configurations of camera 1 and PC 3 in anactivity situation analysis system according to the third embodiment.

According to the third embodiment, position information acquirer 32,position information storage 33, and activity information acquirer 34,which, according to the first embodiment, are provided in PC 3, areprovided in camera 1. Imaging unit 91 that has imaging elements andoptical elements is provided in camera 1, and an image that is outputfrom imaging unit 91 is input into position information acquirer 32. Onthe other hand, activity information storage 35,observation-period-of-time controller 36, activity informationaggregator 37, activity map image generator 38, output controller 39,and processing condition setter 40 are provided in PC 3.

In each unit of camera 1 and PC 3, the same processing as in the firstembodiment is performed. However, particularly, according to the thirdembodiment, the pieces of information that are set in processingcondition setter 40, that is, the pieces of information relating to theobservation area, the unit time, and the like are sent from controller93 of PC 3 to controller 92 of camera 1. Based on these pieces ofinformation, predetermined processing is performed in each of positioninformation acquirer 32 and activity information acquirer 34. It ispossible to make various changes to a configuration in which constituentelements are mounted in camera 1 and PC 3. Most of the constituentelements are mounted in camera 1. Only the processing condition settingand output control of the monitoring moving image may be set to beperformed from PC 3 using a browser.

The present invention is described above based on the specificembodiments, but these embodiments are only examples, and the presentinvention is not limited by these embodiments. Furthermore, allconstituent elements of each of the activity situation analysisapparatus, the activity situation analysis system, and the activitysituation analysis method according to the embodiments described above,of the present invention, are not necessarily essential, and it ispossible to make a selection from among the all constituent elementswithin a range that does not depart from the scope of the presentinvention.

For example, according to the embodiments described above, an example ofa retail store such as a convenience store is described, but the presentinvention is not limited to this retail store, and is also applicable tostores that are different from the retail store in terms of types ofbusiness, for example, such as a restaurant and a bank. Additionally, itis also possible to apply the present invention to the use in monitoringareas other than the store.

Furthermore, according to the embodiments, an example is described abovein which the moving object that is a target for the activity situationanalysis processing is defined as a person, but moving objects otherthan the person, for example, vehicles such as an automobile and abicycle, may be targets for the activity situation analysis processing.

Furthermore, according to the embodiments described above, asillustrated in FIG. 2, camera 1 is set to be an omnidirectional camerawith a fish-eye lens that has a photographing range of 360 degrees, butit is also possible to adopt a camera with a predetermined angle ofview, which is called a box camera.

Furthermore, according to the embodiments described above, asillustrated in FIG. 9 and other figures, the display point in time ofthe moving image and the length and the range (the points in time of thestarting point and of the ending point of the observation period oftime) of the observation period of time are set to be designated bysliders 64, 65 a, and 65 b, but the display forms of sliders 64, 65 a,and 65 b are not limited to the examples that are illustrated in FIG. 9and other figures. Various display forms can be employed. Additionally,in addition to being designated by the sliders, the display point intime and the length and the range of the observation period of time maybe set to be designated by a selection from a selection menu or bydirect input of a numerical value.

Furthermore, according to the embodiments described above, asillustrated in FIGS. 3A, 3B, 16A, and 16B, the monitoring moving imagethat results from superimposing the activity map image onto the movingimage of the monitoring area is set to be displayed, but the activitymap image may not be superimposed onto the moving image of themonitoring area. That is, the activity map image and the moving image ofthe monitoring area may be set to be displayed side by side on themonitoring screen.

Furthermore, according to the embodiments described above, the pieces ofstaying information (the staying time and the number of stays (thenumber of staying persons)) indicating the staying situation of themoving object are set to be acquired as the activity informationindicating the activity situation of the moving object in the monitoringarea, but the activity information is not limited to the pieces ofstaying information. Pieces of information indicating various degrees ofactivity can be acquired and thus the activity map image can begenerated. Furthermore, both of the staying time and the number of staysas the activity information are set to be acquired, and the activity mapimage that expresses both of the staying time and the number of stays isset to be displayed, but only one of the staying time and the number ofstays may be set to be acquired and the activity map image thatexpresses only one of the staying time and the number of stays may beset to be displayed.

Furthermore, according to the embodiments described above, the motionline of the person is set to be acquired and the pieces of activityinformation (the staying time and the number of stays) are set to beacquired based on the motion line, but this activity informationacquirement processing is not limited to the activity information thatis based on the motion line. For example, coordinate informationrelating to a person frame (a rectangular area in which a person ispresent) may be set to be acquired as the position information on theperson, the number of times that every pixel (a detection element) ispositioned within the person frame may be set to be counted, a countvalue of every pixel may be set to be acquired, suitable statisticalprocessing, for example, averaging, may be set to be performed on thecount value of every pixel for aggregation in units of observation areas(refer to FIGS. 3A and 3B) or in units of grids (refer to FIGS. 16A and16B), and thus the activity information on every observation area or onevery grid may be set to be acquired. Furthermore, coordinateinformation relating to a central point of the person frame may be setto be acquired, the number of times that the central point of the personframe is positioned within the observation area or the grid may be setto be counted, and thus the activity information on every observationarea or on every grid may be set to be acquired.

Furthermore, according to the first embodiment, PC 3 that is provided inthe store is set to be caused to perform the processing necessary forthe activity situation analysis, but PC 11 that is provided in the headoffice or cloud computer 12 that makes up a cloud computing system, asillustrated in FIG. 1, may be set to be caused to perform the necessaryprocessing. Furthermore, the necessary processing may be shared amongmultiple information processing apparatuses, and information may be setto be received in multiple information processing apparatuses over acommunication medium such as an IP network or a LAN. In this case, theactivity situation analysis system is configured from the multipleinformation processing apparatuses that share the necessary processing.

With this configuration, among processing operations necessary for theactivity situation analysis, an apparatus that is provided in the storesuch as PC 3 or camera 1 may be set to be caused to perform processingthat requires at least a large amount of computing, for example, persondetection processing. With this configuration, because an amount ofnecessary information data is reduced in remaining processing, even ifan information processing apparatus that is installed in a place otherthan the store, for example, PC 11 that is provided in the head officeis set to be caused to perform the remaining processing, communicationload can be reduced. Because of this, management of the system using awide area network connection type is made easy.

Furthermore, among the processing operations necessary for the activitysituation analysis, cloud computer 12 may be set to be caused to performthe processing that requires at least a large amount of computing, forexample, the person detection processing. With this configuration, anamount of computing is reduced in the remaining processing, a high-speedinformation processing apparatus is unnecessary on the user side such asthe store, and a user-borne cost can be reduced.

Furthermore, cloud computer 12 may be set to be caused to perform allnecessary processing operations, or among the necessary processingoperations, cloud computer 12 may be set to be caused to share at leastimage output processing. With this configuration, in addition to PCs 3and 11 that are provided in the store and the head office, respectively,an image can be displayed on a portable terminal such as smart phone 13or tablet terminal 14, and thus a situation that occurs within the storecan be checked in an arbitrary place such as a visiting destinationother than the store or the head office.

Furthermore, according to the present embodiment, a case where PC 3 thatis installed in the store is caused to perform the processing necessaryfor the activity situation analysis, the monitoring screen or theprocessing condition setting screen is caused to be displayed on monitor7 of PC 3, and necessary input and output are performed on PC 3 isdescribed, but the necessary input and output may be set to be performedin an information processing apparatus that is separate from theinformation processing apparatus that performs the processing necessaryfor the activity situation analysis, for example, in a portable terminalsuch as PC 11 installed in the head office or tablet PC 14.

The activity situation analysis apparatus, the activity situationanalysis system, and the activity situation analysis method according tothe present invention, are useful as the activity situation analysisapparatus, the activity situation analysis system, and the activitysituation analysis method, respectively, each of which enables the userto easily know how the activity situation of the moving object changeswhile the moving image of the monitoring area is displayed, has aneffect that the customization can be easily performed which variouslychanges the analysis form of the activity situation according to theuser need, observes the activity situation of the moving object in themonitoring area, generates the activity map image that results fromvisualizing the activity situation of the moving object, based on aresult of the observation, and displays the generated activity map imageon the display device.

What is claimed is:
 1. An activity situation analysis apparatus thatobserves an activity situation of a moving object in a monitoring area,generates an activity map image that results from visualizing theactivity situation of the moving object based on a result of theobservation, and displays the generated activity map image on a displaydevice, the activity situation analysis apparatus comprising: a memoryin which an instruction is stored; a processor that detects the movingobject from a moving image of the monitoring area and acquires positioninformation on a plurality of moving objects including the movingobject; acquires activity information based on the position informationon the plurality of moving objects; accepts setting conditions relatingto an observation period of time for observing activity situation of themoving object whether the observation period of time is fixed or theobservation period of time is changed in such a manner that, accordingto a progress of a display point in time of the moving image of themonitoring area, the observation period of time moves following thedisplay point in time; controls the observation period of time inaccordance with the conditions relating to the observation period oftime that is set; aggregates the activity information based on theobservation period of time that is controlled, and acquires the activityinformation during the observation period of time; generates theactivity map image based on the activity information during theobservation period of time, the activity map image includes a firstindicator to indicate a stay time and a second indicator to indicate anumber of stays, and an area of the first indicator and an area of thesecond indicator overlap at least in part to provide a combination ofinformation pertaining to the first indicator and the second indicator;generates a monitoring moving image including the activity map image andthe moving image of the monitoring area at every predetermined point intime, and outputs the generated monitoring moving image to the displaydevice; after the monitoring moving image is outputted, modifies theobservation period of time to provide a new observation period of time;acquires additional activity information during a difference periodbetween the observation period of time and the new observation period oftime; and update the activity information with the additional activityinformation acquired during the difference period to reflect combinedactivity information during the observation period of time and thedifference period without reprocessing the activity information of theobservation period of time, wherein, in a case where the processoraccepts setting one of the conditions as the observation period of timebeing fixed, the activity map image included in the monitoring movingimage is not changed at every predetermined point in time, and whereinone of the first indicator and the second indicator is displayed as ageometric shape of varying size, and other of the first indicator andthe second indicator is displayed as different shades of a color.
 2. Theactivity situation analysis apparatus of claim 1, wherein the processorgenerates the monitoring moving image that results from superimposingthe activity map image onto the moving image of the monitoring area, andoutputs the generated monitoring moving image.
 3. The activity situationanalysis apparatus of claim 1, wherein the processor controls theobservation period of time in such a manner that, according to theprogress of the display point in time of the moving image of themonitoring area, the observation period of time moves following thedisplay point in time.
 4. The activity situation analysis apparatus ofclaim 3, wherein the processor controls the observation period of timein such a manner that, based on a setting length of the observationperiod of time that is set, a period of time that is set by the settinglength of the observation period of time earlier than the display pointin time of the moving image of the monitoring area is the observationperiod of time.
 5. The activity situation analysis apparatus of claim 4,wherein the processor controls the observation period of time in such amanner that a period of time from a display starting point in time tothe display point in time is the observation period of time until anelapsed period of time from the display starting point in time of themoving image of the monitoring area to the display point in time reachesthe setting length of the observation period of time.
 6. The activitysituation analysis apparatus of claim 1, wherein the processor controlsthe observation period of time in such a manner that, based on a settingrange of the observation period of time, which is set, within thesetting range of the observation period of time, a period of time fromthe display starting point in time of the moving image of the monitoringarea, which is a starting point of the observation period of time, to adisplay point in time is the observation period of time.
 7. The activitysituation analysis apparatus of claim 1, wherein the processor sets alength of or a range of the observation period of time according to theuser input operation that arbitrarily designates the length of or therange of the observation period of time as conditions of the observationperiod of time.
 8. The activity situation analysis apparatus of claim 1,wherein the processor generates the monitoring moving image includingthe activity map image and a current moving image of the monitoring areain real time, and outputs the generated monitoring moving image.
 9. Theactivity situation analysis apparatus of claim 1, wherein the processorfurther sets a unit time relating to the acquisition of the activityinformation according to a user input operation, and wherein theprocessor acquires the activity information on every unit time based onthe unit time that is set.
 10. The activity situation analysis apparatusof claim 1, wherein the activity information includes informationrelating to at least either of a number of stays that is a number ofmoving objects that stay in an observation area that is set to be withinthe monitoring area and a staying time that is a time for which themoving object stays within the observation area.
 11. The activitysituation analysis apparatus of claim 1, wherein the processor acquiresthe activity information based on the position information of theplurality of moving objects per a unit time, and wherein the processoracquires the activity information during the observation period of timeby aggregating the activity information acquired for units of timecorresponding to the observation period of time.
 12. The activitysituation analysis apparatus of claim 1, wherein, in a case where theprocessor accepts setting one of the conditions as the observationperiod of time being changed, the activity map image included in themonitoring moving image is updated at every predetermined point in time,and wherein, when the observation period of time is changed, anobservation duration remains same during observation, but correspondingstart time and end time are updated at the every predetermined point intime and are not predetermined.
 13. The activity situation analysisapparatus of claim 12, wherein a first observation period of time and asecond observation period of time partially overlap one another, suchthat a start time of the second observation period is during the firstobservation period.
 14. An activity situation analysis system thatobserves an activity situation of a moving object in a monitoring area,generates an activity map image that results from visualizing theactivity situation of the moving object based on a result of theobservation, and displays the generated activity map image on a displaydevice, the activity situation analysis system comprising: a camera thatcaptures a moving image of the monitoring area; and multiple informationprocessing apparatuses, wherein any one of the multiple informationprocessing apparatuses includes a memory in which an instruction isstored, and a processor that: detects the moving object from the movingimage of the monitoring area and acquires position information on aplurality of moving objects including the moving object, acquiresactivity information based on the position information on the pluralityof moving objects, accepts setting conditions relating to an observationperiod of time for observing activity situation of the moving objectwhether the observation period of time is fixed or the observationperiod of time is changed in such a manner that, according to a progressof a display point in time of the moving image of the monitoring area,the observation period of time moves following the display point intime, controls the observation period of time in accordance with theconditions of the observation period of time that is set, aggregates theactivity information based on the observation period of time that iscontrolled, and acquires the activity information during the observationperiod of time, generates the activity map image based on the activityinformation during the observation period of time, the activity mapimage includes a first indicator to indicate a stay time and a secondindicator to indicate a number of stays, and an area of the firstindicator and an area of the second indicator overlap at least in partto provide a combination of information pertaining to the firstindicator and the second indicator, generates a monitoring moving imageincluding the activity map image and the moving image of the monitoringarea at every predetermined point in time, and outputs the generatedmonitoring moving image to the display device, after the monitoringmoving image is outputted, modifies the observation period of time toprovide a new observation period of time, acquires additional activityinformation during a difference period between the observation period oftime and the new observation period of time, and update the activityinformation with the additional activity information acquired during thedifference period to reflect combined activity information during theobservation period of time and the difference period withoutreprocessing the activity information of the observation period of time,wherein, in a case where the processor accepts setting one of theconditions as the observation period of time being fixed, the activitymap image included in the monitoring moving image is not changed atevery predetermined point in time, and wherein one of the firstindicator and the second indicator is displayed as a geometric shape ofvarying size, and other of the first indicator and the second indicatoris displayed as different shades of a color.
 15. An activity situationanalysis system that observes an activity situation of a moving objectin a monitoring area, generates an activity map image that results fromvisualizing the activity situation of the moving object based on aresult of the observation, and displays the generated activity map imageon a display device, the activity situation analysis system comprising:a camera that captures a moving image of the monitoring area; and aninformation processing apparatus, wherein any one of the camera and theinformation processing apparatus includes, a processor that detects themoving object from the moving image of the monitoring area and acquiresposition information on a plurality of moving objects including themoving object, acquires activity information based on the positioninformation on the plurality of moving objects, accepts settingconditions relating to an observation period of time for observingactivity situation of the moving object whether the observation periodof time is fixed or the observation period of time is changed in such amanner that, according to a progress of a display point in time of themoving image of the monitoring area, the observation period of timemoves following the display point in time, controls the observationperiod of time in accordance with the conditions of the observationperiod of time that is set, aggregates the activity information based onthe observation period of time that is controlled, and acquires theactivity information during the observation period of time, generatesthe activity map image based on the activity information during theobservation period of time, the activity map image includes a firstindicator to indicate a stay time and a second indicator to indicate anumber of stays, and an area of the first indicator and an area of thesecond indicator overlap at least in part to provide a combination ofinformation pertaining to the first indicator and the second indicator,generates a monitoring moving image including the activity map image andthe moving image of the monitoring area at every predetermined point intime, and outputs the generated monitoring moving image to the displaydevice, after the monitoring moving image is outputted, modifies theobservation period of time to provide a new observation period of time,acquires additional activity information during a difference periodbetween the observation period of time and the new observation period oftime, and update the activity information with the additional activityinformation acquired during the difference period to reflect combinedactivity information during the observation period of time and thedifference period without reprocessing the activity information of theobservation period of time, wherein, in a case where the processoraccepts setting one of the conditions as the observation period of timebeing fixed, the activity map image included in the monitoring movingimage is not changed at every predetermined point in time, and whereinone of the first indicator and the second indicator is displayed as ageometric shape of varying size, and other of the first indicator andthe second indicator is displayed as different shades of a color.
 16. Anactivity situation analysis method of causing an information processingapparatus to perform processing that observes an activity situation of amoving object in a monitoring area, generates an activity map image thatresults from visualizing the activity situation of the moving objectbased on a result of the observation, and displays the generatedactivity map image on a display device, wherein an instruction that isexecuted by a processor of the information processing apparatus and thatis stored in a memory includes, detecting the moving object from amoving image of the monitoring area and acquiring position informationon a plurality of moving objects including the moving object, acquiringactivity information based on the position information on the pluralityof moving objects, accepting setting conditions relating to anobservation period of time for observing activity situation of themoving object whether the observation period of time is fixed or theobservation period of time is changed in such a manner that, accordingto a progress of a display point in time of the moving image of themonitoring area, the observation period of time moves following thedisplay point in time, controlling the observation period of time inaccordance with the conditions relating to the observation period oftime that is set, aggregating the activity information based on theobservation period of time that is controlled and acquiring the activityinformation during the observation period of time, generating theactivity map image based on the activity information during theobservation period of time, the activity map image includes a firstindicator to indicate a stay time and a second indicator to indicate anumber of stays, and an area of the first indicator and an area of thesecond indicator overlap at least in part to provide a combination ofinformation pertaining to the first indicator and the second indicator,generating a monitoring moving image including the activity map imageand the moving image of the monitoring area at every predetermined pointin time and outputting the generated monitoring moving image to thedisplay device, modifying, after the monitoring moving image isoutputted, the observation period of time to provide a new observationperiod of time, acquiring additional activity information during adifference period between the observation period of time and the newobservation period of time, and updating the activity information withthe additional activity information acquired during the differenceperiod to reflect combined activity information during the observationperiod of time and the difference period without reprocessing theactivity information of the observation period of time, wherein, in acase where the processor accepts setting one of the conditions as theobservation period of time being fixed, the activity map image includedin the monitoring moving image is not changed at every predeterminedpoint in time, and wherein one of the first indicator and the secondindicator is displayed as a geometric shape of varying size, and otherof the first indicator and the second indicator is displayed asdifferent shades of a color.